Papers with mixing mechanisms
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Sparse Mixer encoder model outperforms BERT on GLUE and SuperGLUE, trains 65% faster and runs inference 61% faster. |
| Approach: | They combine the capacity of sparsely gated Mixture-of-Experts (MoE) with the speed and stability of linear, mixing transformations to design the Sparse Mixer encoder model. |
| Outcome: | The proposed model outperforms BERT on GLUE and SuperGLUE but trains and runs twice as fast. |